Skip to content

feat(extras): rely on MicroVM auto-resume - #913

Merged
ParidelPooya merged 5 commits into
mainfrom
feat/microvm-rely-on-auto-resume
Nov 9, 2026
Merged

ParidelPooya merged 5 commits into
mainfrom
feat/microvm-rely-on-auto-resume

Conversation

@ParidelPooya

@ParidelPooya ParidelPooya commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The service resumes a suspended MicroVM on the next request when its idle policy sets autoResumeEnabled. So the MicroVM operations no longer resume a MicroVM themselves. The SDK never calls ResumeMicrovm, and the function's role no longer needs lambda:ResumeMicrovm.

This change builds on #910, now on main. It keeps the order of the state check from #910: after a sign that the MicroVM does not run, the state check runs first, and only the invocation reserve limits it. It keeps the one internal denial error, MicrovmPermissionDeniedError. Only the GetMicrovm case remains, so its message names lambda:GetMicrovm.

Twin PR: aws/aws-durable-execution-sdk-js#1182

Design note (CONTRIBUTING rule 3)

1. Problem

The MicroVM operations resume a suspended MicroVM themselves: GetMicrovm, ResumeMicrovm, polling until RUNNING, a recorded resume mode, and a "may be suspended" mark after suspend_after. This is a large share of the code and of the review rounds, and it needs lambda:ResumeMicrovm in the function role. The service can do the same job.

2. Cause

The SDK was built before IdlePolicy.autoResumeEnabled was known and tested. Without it, a suspended MicroVM's endpoint answers 502 MICROVM_CONNECT_FAILED forever, so the SDK had to resume.

3. What a live probe showed

A probe ran 16 MicroVMs in us-east-1. Every MicroVM was terminated afterwards.

Finding Value
A request to a SUSPENDED MicroVM with autoResumeEnabled: true resumes it Yes. The endpoint holds the request about 750 ms (720 to 1,335 ms, 12 samples), then the worker answers.
Window right after SuspendMicrovm For 4.4 to 4.8 s, a request gets 502 MICROVM_CONNECT_FAILED in about 200 ms and does not resume the MicroVM. GetMicrovm already says SUSPENDED.
During the suspend hook 502 INTERNAL_ERROR, 502 MICROVM_CONNECT_FAILED, or no answer. A request with no answer reaches the worker after the next resume; the worker dedupes by callback ID.
autoResumeEnabled: false, or no idle policy 502 MICROVM_CONNECT_FAILED forever; stays SUSPENDED
Idle policy fields All three required. maxIdleDurationSeconds >= 60, max 2,147,483,647.
Idle timer Counts only inbound requests. It suspended a running job after 66 s.
Worker hooks resume hook fires on every auto-resume, before the request reaches the worker. The worker's 503 refusal never reached a client after an auto-resume.

The current request retries (250 ms doubling to 4 s, 60 s window) reach 7.75 s after 5 retries, so they already cover the 5 s window. With a retry_window under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds.

4. Options

Option Result
A. Keep the SDK resume logic Works with or without the policy. Keeps the code, the permission and the review cost.
B. Simplify: rely on autoResumeEnabled, configured by the developer. Chosen. Removes the resume code and lambda:ResumeMicrovm. A developer who suspends must set the policy.
C. The SDK sets the idle policy itself Hides a service setting, and must pick maxIdleDurationSeconds; a wrong value suspends long jobs. The developer knows their job length.
D. B plus a check that reads idlePolicy from GetMicrovm and fails fast when auto-resume is off A guard for a developer mistake. Rejected (no guards; the guideline is documented).

5. Chosen option

  1. The SDK never calls ResumeMicrovm.
  2. The state check from fix(extras): fail microvm() fast on a dead MicroVM #910 stays, only to find a dead MicroVM. After a 502 MICROVM_CONNECT_FAILED or a request without an answer, the request step makes one GetMicrovm call, at most once per 2 s. The call runs first, before the retry window and the backoff, and only the invocation reserve limits it. With less than 1 s left before the reserve, no call starts, and the attempt ends with the retryable MicrovmEndpointUnavailableError.
    • TERMINATING, TERMINATED, not found: fail at once with MicrovmNotRunningError.
    • RUNNING, PENDING, SUSPENDING, SUSPENDED: keep retrying. The service resumes the MicroVM on a request.
    • AccessDenied: fail at once with MicrovmPermissionDeniedError (fix(extras): fail microvm() fast on a dead MicroVM #910 behavior). Its message names lambda:GetMicrovm.
    • Other errors: one warning per step attempt, keep retrying.
  3. suspend_after (the function calls SuspendMicrovm) and the worker's auto_suspend_on_idle stay. Both require the developer's idle policy with autoResumeEnabled: true.
  4. No check that the policy is set. A developer who suspends without it gets 502 until the retries end: about 6 minutes with the default strategy, then MicrovmDeliveryError.
  5. Docs (README, docstrings of suspend_after, auto_suspend_on_idle, idle_policy):
    • "To suspend a MicroVM, set idle_policy with "autoResumeEnabled": True. The SDK does not resume a MicroVM; the service resumes it on the next request."
    • "Set maxIdleDurationSeconds above maximumDurationInSeconds (the session timeout plus 5 minutes). The idle timer counts only inbound requests, so a shorter value suspends a running job."
    • "Set suspendedDurationSeconds to the session timeout or more." What happens to a job when that time ends is not verified.
    • "Without autoResumeEnabled, a job after a suspend fails after its retries with MicrovmDeliveryError."
    • "A job that arrives more than about 5 seconds after a suspend takes about 1 s longer."
    • "With a retry_window under about 9 seconds, the step retry strategy covers the rest of that window, at the cost of one or two more step attempts. A job sent right after a suspend can take up to about 10 seconds."
    • Permissions: the function role no longer needs lambda:ResumeMicrovm. It needs lambda:GetMicrovm for HTTP delivery, and lambda:SuspendMicrovm for suspend_after.

6. What it removes

  • ensure_running, its loop _wait_until_running, EnsureRunningOptions (lifecycle.py).
  • The SUSPENDED/SUSPENDING branch and the ResumeMicrovm call in resume_if_suspended. It is renamed check_not_ended, and it takes the client, the MicroVM ID, and the remaining-time function.
  • The ResumeMicrovm case of MicrovmPermissionDeniedError.
  • The resume-wait cap from fix(extras): fail microvm() fast on a dead MicroVM #910 (min(max_wait_seconds, retry_window_seconds)). The check no longer waits, so no wait remains to cap.
  • The recorded resume mode: resume_before_delivery in LaunchResult and resume_from_config (operation.py, session.py). LaunchResult.from_dict ignores a resumeBeforeDelivery key, so an old checkpoint still replays.
  • The _may_be_suspended mark and _resume_mode of MicrovmSession.
  • The recheck path of send_job (request.py), and the resume argument of deliver. SendJobOptions.suspend_check is renamed state_check, and it takes no argument.
  • SendJobOptions.window_started_at. With no check before delivery, it equaled the start of send_job.
  • ResumeMicrovm from MicrovmApiOperation and from microvm_not_found.
  • lambda:ResumeMicrovm from the README permissions and the example docs.

Stays: the request retries, the dead-MicroVM check and its order, MicrovmPermissionDeniedError, suspend_after, worker auto_suspend_on_idle, the worker's suspend/resume hooks, its 503 refusal and 30 s grace (still needed for the worker's own SuspendMicrovm), and its callback-ID dedupe (now also needed for late delivery after a frozen request).

7. Replay and compatibility

  • Operation names and subtypes do not change.
  • An execution in flight that launched in resume mode replays its launch checkpoint with resumeBeforeDelivery: true. The new code ignores the field. Its next job no longer resumes the MicroVM. It works only if that MicroVM was launched with autoResumeEnabled: true. A suspend_after session without an idle policy is affected too: its next job after the suspend no longer resumes the MicroVM and fails after its retries with MicrovmDeliveryError. The README says both. The package is unpublished.
  • The two suspend examples and their cloud tests set the idle policy.

Source lines (aws-durable-execution-sdk-python-extras/src only)

Lines
Removed 635
Added 245
Net -390

Coverage counts 967 statements and 188 branches, down from 1,094 and 228 on main.

Tests

Check Result
Extras unit tests, coverage 436 passed, 100% statements and branches (main: 490 passed, 100%)
Worker unit tests 473 passed
Examples tests 204 passed, 6 skipped (cloud-only)
mypy, extras src and tests, worker src and tests clean
hatch fmt --check no findings in the changed files; the linter and the formatter pass on every changed .py file
Cloud tests run by CI

New and changed tests:

Test What it proves
conftest.py fake resume_microvm raises, and the autouse fixture no_resume_microvm fails any test whose fake client recorded the call No code path calls ResumeMicrovm
lifecycle_test.py::test_check_returns_the_endpoint_of_a_microvm_that_has_not_ended[RUNNING, PENDING, SUSPENDING, SUSPENDED] One GetMicrovm call, no error, no resume
lifecycle_test.py::test_check_raises_the_state_error_for_a_microvm_that_ends[TERMINATING, TERMINATED], ..._for_a_removed_microvm Fail fast
lifecycle_test.py::test_check_fails_at_once_when_get_microvm_is_denied MicrovmPermissionDeniedError, message names lambda:GetMicrovm
operation_test.py::test_checks_the_state_after_a_sign_that_uses_up_the_retry_window[connect-failed, no-answer] (from #910) The check runs first after a sign, so a 1-second window still gets it
operation_test.py::test_ends_an_attempt_of_a_suspended_microvm_without_a_deadline Replaces the resume-wait cap test of #910. Without a Lambda deadline, a SUSPENDED MicroVM whose endpoint keeps answering 502 gets one GetMicrovm call, no ResumeMicrovm, and the window ends the attempt
operation_test.py::test_a_custom_strategy_sees_the_denial_error_type A custom strategy sees MicrovmPermissionDeniedError for a denied GetMicrovm
request_test.py::test_runs_the_state_check_when_the_sign_used_up_the_window With a 0-second window, the check still runs before the window ends the tier
request_test.py::test_starts_no_state_check_within_1_second_of_the_reserve The reserve still applies: no GetMicrovm call
operation_test.py::test_keeps_sending_the_job_to_a_microvm_suspended_from_outside[SUSPENDING, SUSPENDED] microvm keeps retrying
session_test.py::test_keeps_sending_to_a_microvm_that_the_worker_suspended_and_uses_its_endpoint[...], test_keeps_sending_the_job_after_a_suspend_after_job[...], test_a_job_after_a_replayed_suspend_after_job_keeps_sending, test_runs_both_items_when_both_find_the_microvm_suspended Session jobs keep retrying after either suspend, also after replay and in a map
session_test.py::test_replays_a_launch_checkpoint_with_the_resume_mode_of_an_older_release A launch checkpoint with resumeBeforeDelivery: true replays, and the next job works
operation_test.py::test_launch_result_ignores_the_resume_mode_of_an_older_release from_dict ignores the field, to_dict does not write it
session_test.py::test_writes_no_resume_mode_in_the_launch_checkpoint[...] New launches write no resume mode
worker_composed_test.py (3 tests) With the real worker, a stand-in service answers the first request after a suspend with 502, then auto-resumes on the next request with the resume hook. Jobs succeed without ResumeMicrovm, with and without the resume hook, and when a resend meets a suspend.

Removed tests. The behaviors they covered no longer exist:

  • Every ensure_running test, and the resume, resume-conflict and resume-wait tests of the state check (lifecycle_test.py).
  • From fix(extras): fail microvm() fast on a dead MicroVM #910: lifecycle_test.py::test_check_fails_at_once_when_resume_microvm_is_denied, lifecycle_test.py::test_check_names_no_request_id_for_an_error_of_another_call (the ResumeMicrovm case; the GetMicrovm case stays), operation_test.py::test_the_wait_for_a_resume_gets_a_new_retry_window_without_a_deadline (replaced, see above), and the resume case of operation_test.py::test_a_custom_strategy_sees_one_error_type_for_a_denial.
  • request_test.py::test_counts_the_window_from_the_given_start, with window_started_at.
  • The resume-mode recording and deploy tests, the recheck tests of send_job, the "flag" tests of suspend_after, and the ResumeMicrovm scenarios of request_ids_test.py.

Edited expectations (rule 5.4):

  • Tests that counted a GetMicrovm call before delivery now expect none, because no check runs before delivery.
  • Tests that expected a dead MicroVM to fail before any request now send one request that gets 502 MICROVM_CONNECT_FAILED first, because the check runs only after that sign.
  • request_ids_test.py: "GetMicrovm denies access" expects MicrovmPermissionDeniedError with its message, because the strict pre-check that raised the raw AccessDeniedException is gone. The message-based retry strategy case for GetMicrovm is removed, because a throttled state check logs a warning and never reaches the strategy.
  • request_test.py::test_checks_the_state_after_a_connect_failure_or_no_answer: the check takes no argument, so the test records when it runs (0 and 3.75 s, before the delay) instead of the time it received.
  • The warning text changes from "check whether the MicroVM is suspended" to "check whether the MicroVM has ended". The SuspendMicrovm warning no longer says "The next job checks its state."
  • The unavailable message of a stalled GetMicrovm changes from "did not reach RUNNING" to "did not end before the invocation reserve", because the check no longer waits for RUNNING.

Mutation checks (rule 4)

Each row changes one branch in src, runs the extras tests, and restores the file.

Mutation Result
M1 SUSPENDED/SUSPENDING fail fast killed
M2 the state check calls ResumeMicrovm for SUSPENDED killed (fake raises, autouse fixture fails)
M3 the state check runs after the window exit killed
M4 a denied GetMicrovm is treated as transient killed
M5 TERMINATING/TERMINATED no longer fail fast killed
M6 not found no longer fails fast killed
M7 the window limits the state check killed
M8 no reserve guard before the state check killed
M9 the warning repeats on each check killed
M10 no state check after MICROVM_CONNECT_FAILED killed
M11 no state check after a request without an answer killed
M12 no 2 s spacing between checks killed
M13 the endpoint that the check returns is ignored killed
M14 from_dict rejects a resumeBeforeDelivery key killed
M15 to_dict writes resumeBeforeDelivery again killed

Assumptions (rule 6)

  • Auto-resume, the 4.4 to 4.8 s window after SuspendMicrovm, the about 750 ms resume latency, and the idle timer that counts only inbound requests: verified by the live probe above.
  • Auto-resume after the worker's own SuspendMicrovm call from inside the MicroVM: unverified. It is the same API call, so the same behavior is expected.
  • About 6 minutes until MicrovmDeliveryError without autoResumeEnabled: computed from the default retry strategy, not measured.

Decisions not covered

  • Whether outbound traffic, such as heartbeats, resets the idle timer. Not needed when maxIdleDurationSeconds is above the maximum duration.
  • The behavior of suspendedDurationSeconds: 0.
  • Whether to keep auto_suspend_on_idle at all. The service's idle timer with a long job is unsafe, and the worker's own suspend needs the same policy. It stays for now.
  • The lambda:ResumeMicrovm grant in the role that the cloud tests use is defined outside this repository. Remove it in a later change, after both SDKs ship.
  • Documentation-only changes in the worker: the docstrings and comments of worker.py, the docstring of SUSPEND_GRACE_SECONDS in self_suspend.py, two test comments, and the worker README now say that the durable function delivers the job again and the service resumes the MicroVM. The worker code is unchanged.
  • SendJobOptions.suspend_check is renamed state_check, and resume_if_suspended is renamed check_not_ended with a new signature (client, MicroVM ID, remaining-time function).
  • The message of a state check that reaches the invocation reserve is now GetMicrovm for MicroVM <id> did not end before the invocation reserve, after <n> ms. The JS twin uses the same message.
  • No live run of this change outside CI. CI runs the cloud examples.

The service resumes a suspended MicroVM on the next request when
its idle policy sets autoResumeEnabled. So the SDK no longer
resumes a MicroVM itself.

- Remove ensure_running, its poll loop and EnsureRunningOptions.
- Rename resume_if_suspended to check_not_ended. It makes one
  GetMicrovm call after a 502 MICROVM_CONNECT_FAILED or a request
  without an answer. TERMINATING, TERMINATED and not found fail at
  once. A denied call fails at once. Any other state keeps the job
  retrying. Any other error logs one warning per step attempt.
- Remove MicrovmResumeDeniedError and the ResumeMicrovm call.
- Stop writing resumeBeforeDelivery in the launch checkpoint.
  from_dict ignores the field, so an old checkpoint still replays.
- Remove the in-memory suspend mark of a session handle and the
  recheck path of send_job.

The test fake now fails any test that calls ResumeMicrovm.
A session that suspends its MicroVM must set an idle policy with
autoResumeEnabled. The README now says so, with the idle timer
rule for maxIdleDurationSeconds, the failure without the policy,
the extra second of the first job after a suspend, and the replay
of a launch checkpoint from the earlier release.

- Remove lambda:ResumeMicrovm from the permissions.
- Add a section on the state check, and remove the in-memory
  suspend record and the resume steps.
- Update the worker README and one worker docstring: the service
  resumes the MicroVM, not the durable function.
The SDK no longer resumes a suspended MicroVM. So the two examples
that suspend their MicroVM set an idle policy with
autoResumeEnabled, a maxIdleDurationSeconds above the MicroVM's
maximum duration, and a suspendedDurationSeconds of the maximum
duration.

The cloud tests keep their SUSPENDED-between-jobs and file-kept
assertions. They now expect a launch checkpoint without
resumeBeforeDelivery. The examples test README drops
lambda:ResumeMicrovm from the role needs.
The rebase onto the dead-MicroVM fix keeps its order: after a sign,
the state check runs first, and only the invocation reserve limits
it. The check no longer waits for a resume. So the test of the
resume-wait cap now proves that an attempt without a deadline ends,
and the denial test covers only GetMicrovm.

The worker README now says that the service resumes the MicroVM
after a 503.
- Say that a job sent right after a suspend can take up to about 10
  seconds. A retry_window under about 9 seconds relies on the step
  retry strategy for the rest of the 5-second window.
- Add a suspend_after session without an idle_policy to the in-flight
  note.
- Place the 30-second worker refusal after the suspend.
- Use the Python names idle_policy and True, and the session timeout.
- Add the suspendedDurationSeconds guideline. Its end is not verified.
- Say that the service resumes the MicroVM in the worker docs.
- Rewrap a long docstring line in an example.
@ParidelPooya ParidelPooya changed the title refactor(extras): rely on MicroVM auto-resume feat(extras): rely on MicroVM auto-resume Nov 7, 2026
@ParidelPooya
ParidelPooya marked this pull request as ready for review November 7, 2026 13:19
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 7, 2026 13:19 — with GitHub Actions Active
@ParidelPooya
ParidelPooya deployed to ai-pr-review-runtime November 7, 2026 13:22 — with GitHub Actions Active
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings, including against CONTRIBUTING rules 1–9. Tests were not executed during this read-only review. Residual risk: the PR explicitly marks auto-resume after worker-initiated suspension as unverified.

Reviewed commit 3f441de62c5ba65ec7ebf33d91de7bf96e180da9. Workflow run

@yaythomas yaythomas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

Compatibility statements and interfaces for earlier preview versions are not necessary while the extras package is alpha.

@ParidelPooya
ParidelPooya merged commit ecbe2cb into main Nov 9, 2026
31 checks passed
@ParidelPooya
ParidelPooya deleted the feat/microvm-rely-on-auto-resume branch November 9, 2026 02:29
ParidelPooya added a commit to aws/aws-durable-execution-sdk-js that referenced this pull request Nov 9, 2026
…VM (#1182)

## Summary

The MicroVM operations no longer resume a suspended MicroVM. The service
resumes a MicroVM that was launched with `idlePolicy.autoResumeEnabled:
true` when the next request arrives. So the SDK never calls
ResumeMicrovm, and the function role no longer needs
`lambda:ResumeMicrovm`.

This branch is rebased on main after #1178. It keeps the state-check
order of #1178: after a sign that the MicroVM does not run, the state
check runs first, limited only by the invocation reserve.

- `ensureRunning`, its polling loop and its options are removed.
- The state check after a sign that the MicroVM does not run is now
`checkNotEnded`. It makes one GetMicrovm call.
- TERMINATING, TERMINATED and not found fail the job at once with
`MicrovmNotRunningError`.
- AccessDenied fails it at once with the internal
`MicrovmPermissionDeniedError`. Only the GetMicrovm case remains, so its
message names `lambda:GetMicrovm`.
- RUNNING, PENDING, SUSPENDING and SUSPENDED keep the request step
retrying.
- Other errors log one warning per step attempt, and the step keeps
retrying.
- The `suspendCheck` option of `sendJob` is now `stateCheck`. The
`recheck` option is removed.
- The resume mode is removed: the session launch no longer writes
`resumeBeforeDelivery`, the `maybeSuspended` mark after a `suspendAfter`
job is gone, and `sendJob` has no `recheck` path.
- `suspendAfter`, `autoSuspendOnIdle`, the worker and its hooks are
unchanged. The worker README and TSDoc now say that the service resumes
the MicroVM.
- Docs, TSDoc, the CHANGELOG, the `microvm-session` example and the
end-to-end scenarios now set or describe the idle policy.

Twin PR:
[aws/aws-durable-execution-sdk-python#913](aws/aws-durable-execution-sdk-python#913)

## Design note (CONTRIBUTING rule 3)

### 1. Problem

The MicroVM operations resume a suspended MicroVM themselves:
GetMicrovm, ResumeMicrovm, polling until RUNNING, a recorded resume
mode, and a "may be suspended" mark after `suspendAfter`. This is a
large share of the code and of the review rounds, and it needs
`lambda:ResumeMicrovm` in the function role. The service can do the same
job.

### 2. Cause

The SDK was built before `IdlePolicy.autoResumeEnabled` was known and
tested. Without it, a suspended MicroVM's endpoint answers `502
MICROVM_CONNECT_FAILED` forever. So the SDK had to resume.

### 3. What a live probe showed (us-east-1, 16 MicroVMs)

| Finding | Value |
|---|---|
| A request to a SUSPENDED MicroVM with `autoResumeEnabled: true`
resumes it | Yes. The endpoint holds the request about 750 ms (720 to
1,335 ms, 12 samples), then the worker answers. |
| Window right after SuspendMicrovm | For 4.4 to 4.8 s, a request gets
`502 MICROVM_CONNECT_FAILED` in about 200 ms and does not resume the
MicroVM. GetMicrovm already says SUSPENDED. |
| During the suspend hook | `502 INTERNAL_ERROR`, `502
MICROVM_CONNECT_FAILED`, or no answer. A request with no answer reaches
the worker after the next resume. The worker dedupes by callback ID. |
| `autoResumeEnabled: false`, or no idle policy | `502
MICROVM_CONNECT_FAILED` forever. The MicroVM stays SUSPENDED. |
| Idle policy fields | All three are required. `maxIdleDurationSeconds`
>= 60, max 2,147,483,647. |
| Idle timer | It counts only inbound requests. It suspended a running
job after 66 s. |
| Worker hooks | The `resume` hook fires on every auto-resume, before
the request reaches the worker. The worker's 503 refusal never reached a
client after an auto-resume. |

The request retries (250 ms doubling to 4 s, 60 s window) reach 7.75 s
after 5 retries. So they already cover the 5 s window. With a
`retryWindow` under about 9 seconds, the step retry strategy covers the
rest of that window, at the cost of one or two more step attempts. A job
sent right after a suspend can take up to about 10 seconds.

### 4. Options

| Option | Result |
|---|---|
| A. Keep the SDK resume logic | Works with or without the policy. Keeps
the code, the permission and the review cost. |
| B. Rely on `autoResumeEnabled`, configured by the developer.
**Chosen.** | Removes the resume code and `lambda:ResumeMicrovm`. A
developer who suspends must set the policy. |
| C. The SDK sets the idle policy itself | Hides a service setting, and
must pick `maxIdleDurationSeconds`. A wrong value suspends long jobs.
The developer knows their job length. |
| D. B plus a check that reads `idlePolicy` from GetMicrovm and fails
fast when auto-resume is off | A guard for a developer mistake.
Rejected: no guards, and the guideline is documented. |

### 5. Chosen option

1. The SDK never calls ResumeMicrovm.
2. The state check from #1178 stays, only to find a dead MicroVM. After
a `502 MICROVM_CONNECT_FAILED` or a request without an answer, it makes
one GetMicrovm call, at most once per 2 s. It runs before the retry
window and the backoff decide, and only the invocation reserve limits it
(#1178).
- TERMINATING, TERMINATED, not found: fail at once with
`MicrovmNotRunningError`.
- RUNNING, PENDING, SUSPENDING, SUSPENDED: keep retrying. The service
resumes the MicroVM on a request.
- AccessDenied: fail at once with `MicrovmPermissionDeniedError`
(#1178).
   - Other errors: one warning per step attempt, keep retrying.
3. `suspendAfter` (the function calls SuspendMicrovm) and the worker's
`autoSuspendOnIdle` stay. Both require the developer's idle policy with
`autoResumeEnabled: true`.
4. No check that the policy is set. A developer who suspends without it
gets 502 until the retries end: about 6 minutes with the default
strategy, then `MicrovmDeliveryError`.
5. Docs (README, TSDoc of `suspendAfter`, `autoSuspendOnIdle`,
`idlePolicy`):
- To suspend a MicroVM, set `idlePolicy` with `autoResumeEnabled: true`.
The SDK does not resume a MicroVM. The service resumes it on the next
request.
- Set `maxIdleDurationSeconds` above `maximumDurationInSeconds` (the
operation timeout plus 5 minutes). The idle timer counts only inbound
requests, so a shorter value suspends a running job.
- Without `autoResumeEnabled`, a job after a suspend fails after its
retries with `MicrovmDeliveryError`.
- A job that arrives more than about 5 seconds after a suspend takes
about 1 s longer. With a `retryWindow` under about 9 seconds, the step
retry strategy covers the rest of that window, at the cost of one or two
more step attempts. A job sent right after a suspend can take up to
about 10 seconds.
- Permissions: the function role no longer needs `lambda:ResumeMicrovm`.
It needs `lambda:GetMicrovm` for HTTP delivery, and
`lambda:SuspendMicrovm` for `suspendAfter`.

### 6. What it removes

- `ensureRunning`, its loop `waitUntilRunning`, `EnsureRunningOptions`
(`lifecycle.ts`).
- The SUSPENDED/SUSPENDING branch and the ResumeMicrovm call in
`resumeIfSuspended`, renamed `checkNotEnded`. With it goes the
ResumeMicrovm case of `MicrovmPermissionDeniedError`.
- The recorded resume mode: `resumeBeforeDelivery` in the launch result
and `resumeMode()` (`session.ts`, `shared.ts`). The launch checkpoint
field stays readable on replay of old executions, but is no longer
written or used.
- The `maybeSuspended` mark and the `job.then(mark, mark)` chain after a
`suspendAfter` job.
- The `recheck` path of `sendJob` (`request.ts`).
- `lambda:ResumeMicrovm` from the README permissions, the end-to-end
role and the examples guide.

Stays: the request retries, the dead-MicroVM check and its order from
#1178, `MicrovmPermissionDeniedError`, `suspendAfter`, worker
`autoSuspendOnIdle`, the worker's `suspend`/`resume` hooks, its 503
refusal and 30 s grace (still needed for the worker's own
SuspendMicrovm), and its callback-ID dedupe (now also needed for late
delivery after a frozen request).

### 7. Replay and compatibility

- Operation names and subtypes do not change. The event signatures of
the `microvm-session` example do not change, so its history file is
unchanged.
- An execution in flight that launched in resume mode replays its launch
checkpoint with `resumeBeforeDelivery: true`. The new code ignores the
field. So its next job no longer resumes the MicroVM. That job works
only if the MicroVM was launched with `autoResumeEnabled: true`.
- An execution in flight whose session used `suspendAfter` without an
idle policy recorded `resumeBeforeDelivery: false`. Its next job after
the suspend no longer resumes the MicroVM and fails after its retries
with `MicrovmDeliveryError`.
- The CHANGELOG and the README say both. The package is experimental
(`0.1.0-alpha.0`).

## Rebase onto main

The branch was first built on the head of #1178 before its last review
round. That round changed the same state check: the check runs first
after a sign, and the two denial errors became one
`MicrovmPermissionDeniedError`. The rebase kept both changes:

- `lifecycle.ts`: kept `checkNotEnded` without the resume branch, and
kept the one error class `MicrovmPermissionDeniedError`. Its TSDoc no
longer mentions ResumeMicrovm.
- `request.ts`: kept the check before the window exit and the backoff,
budgeted only by the invocation reserve, under the name `stateCheck`.
The `recheck` branch is gone.
- `shared.ts`: kept the `stateCheck` closure without `maxWaitMs`, `log`
and `windowStartedAt`.
- Tests: the #1178 tests for the short retry window and the reached
reserve are kept, renamed to `stateCheck`. The denied-ResumeMicrovm case
of the custom-retry-strategy test is dropped. The composed test that
sends a suspend hook before the resend now expects the state check
before the first backoff.

## Tests

| Package | After |
|---|---|
| extras (vitest, `tsc --noEmit` first) | 427 tests, 26 files, all pass
|
| extras coverage | `lifecycle.ts`, `microvm.ts`, `session.ts` 100 %.
`request.ts` 100 % lines and branches. All files 97.67 % lines, 96.26 %
branches. |
| microvm-worker (vitest) | 305, all pass |
| examples (vitest) | 276 pass, 3 skipped (cloud only) |
| tsc (extras, e2e, examples, worker) | clean |
| biome (`npm run lint`) | no findings at error level |

New or changed tests:

| Test | What it proves |
|---|---|
| `check-not-ended.test.ts`: RUNNING, PENDING, SUSPENDING, SUSPENDED |
One GetMicrovm call, the endpoint is returned, no other call. |
| `check-not-ended.test.ts`: TERMINATING, TERMINATED, not found, denied,
transient errors, no time left | Fail fast,
`MicrovmPermissionDeniedError` naming `lambda:GetMicrovm`,
`MicrovmStateCheckError`, the reserve. |
| `request-state-check.test.ts` (from #1178) | The check runs when the
retry window has no time left, and makes no call at the reserve. |
| `microvm-state-check.test.ts`: a delayed 502 or no answer with a 1 s
window (from #1178) | `MicrovmNotRunningError` after one GetMicrovm
call. |
| `microvm-state-check.test.ts`: SUSPENDED or SUSPENDING after a 502 | A
`microvm` job keeps sending and succeeds, with no ResumeMicrovm call. |
| `session-suspend.test.ts`: MicroVM SUSPENDED or SUSPENDING after the
wait | The job gets 502s, one state check, then the fake endpoint
auto-resumes it. No ResumeMicrovm call. |
| `session-suspend.test.ts`: no state call before a job | With
`autoSuspendOnIdle` or an `idlePolicy`, no GetMicrovm call before
delivery. |
| `session-suspend.test.ts`: launch checkpoint | A new launch records no
`resumeBeforeDelivery`. A launch checkpoint of an earlier version with
`resumeBeforeDelivery: true` (and `false`) replays, and no job checks
the state before delivery. |
| `session-suspend-check.test.ts`: the service does not resume | A
MicroVM that stays SUSPENDED fails the job with `MicrovmDeliveryError`
after the retries, with no ResumeMicrovm call. |
| `session-suspend-after.test.ts` | The job after a `suspendAfter` job,
also after replay, after a failed job, and when the job was not awaited,
is delivered with no ResumeMicrovm call. |
| `worker-composed.test.ts` | Against the real worker listener and a
fake service that auto-resumes after a 4.8 s settle window: the 503
refusal, the 30 s fallback without the resume hook, a lost answer, and a
suspend hook before a resend. |
| Fake client | A ResumeMicrovm call records `resume` and throws. So any
such call fails the test. |

Removed tests:

- `lifecycle.test.ts`: every `ensureRunning` test.
- `resume-if-suspended.test.ts`: the whole file. Its GetMicrovm cases
moved to `check-not-ended.test.ts`. The resume, conflict, poll,
wait-budget, ResumeMicrovm-denied, ResumeMicrovm-not-found and
other-ResumeMicrovm-error cases are removed.
- `microvm-state-check.test.ts`: the resume of a MicroVM suspended from
outside, the denied ResumeMicrovm, and the ResumeMicrovm case of the
custom-retry-strategy test.
- `session-suspend.test.ts` and `session-suspend-check.test.ts`: the
resume, conflict, recheck, resume-mode and deploy tests.
- `remaining-time.test.ts`: the recheck and shared-window tests.
- `retry-strategy-message.test.ts`: the tests for a throttled GetMicrovm
in a session. A GetMicrovm error no longer reaches the step retry
strategy, so that case no longer exists.
- `invocation-reserve.test.ts`: the stalled ResumeMicrovm at the
reserve.

## Mutation tests

Each mutation was applied alone, and the extras suite was run.

| Mutation | Result |
|---|---|
| M1 `checkNotEnded` calls ResumeMicrovm for SUSPENDED or SUSPENDING |
Killed, 20 tests fail |
| M2 SUSPENDED fails fast like TERMINATED | Killed, 16 tests fail |
| M3 TERMINATED no longer fails fast | Killed, 8 tests fail |
| M4 The state check runs after the window exit | Killed, 5 tests fail |
| M5 The retry window limits the check's budget | Killed, 2 tests fail |
| M6 A denied GetMicrovm is treated as transient | Killed, 7 tests fail
|
| M7 No 2-second limit between state checks | Killed, 7 tests fail |
| M8 No state check after a 502 with `MICROVM_CONNECT_FAILED` | Killed,
42 tests fail |
| M9 The endpoint that the state check reports is ignored | Killed, 2
tests fail |
| M10 A transient GetMicrovm error fails the step instead of a warning |
Killed, 3 tests fail |
| M11 A launch checkpoint with `resumeBeforeDelivery: true` makes a
state check before delivery | Killed, 1 test fails |
| M12 The launch writes `resumeBeforeDelivery` again | Killed, 4 tests
fail |

## Live check (us-east-1, dev account, on the rebased branch)

The end-to-end function role had `lambda:RunMicrovm`,
`lambda:TerminateMicrovm`, `lambda:GetMicrovm`,
`lambda:CreateMicrovmAuthToken` and `lambda:SuspendMicrovm`, and no
`lambda:ResumeMicrovm`. So any ResumeMicrovm call would have failed the
job.

| Scenario | Result |
|---|---|
| `suspend-after-session`: a `suspendAfter` job, then the next job sent
at once, idle policy with `autoResumeEnabled` | SUCCEEDED in 15 s. The
second job read the value that the first job kept in memory, in the same
process. The resume hook fired once. The first `/recall` request got no
answer within the 10 s request timeout: it arrived during the suspend
hook. The next request resumed the MicroVM and got 202 after 547 ms. |
| `suspend-session`: `autoSuspendOnIdle` with a 10 s idle time, idle
policy with `autoResumeEnabled` | SUCCEEDED in 90 s. States PENDING,
RUNNING, SUSPENDED. The value in memory survived. The resume hook fired
once. |

Every MicroVM ended TERMINATED. ListMicrovms showed none in another
state afterwards.

## Unverified assumptions

- In every live run of `suspend-after-session`, the first request after
the suspend got no answer, not the 502 of the settle window. So the
502-window retry path ran only in unit tests and in the earlier probe.
- "About 6 minutes" until `MicrovmDeliveryError` without
`autoResumeEnabled` is computed from the default strategy (5 attempts,
each up to a 60 s window, plus 2, 4, 8 and 16 s delays). It was not
measured.
- The behavior of `suspendedDurationSeconds` when it expires was not
tested. The docs ask for the session timeout or more.
- Whether outbound traffic, such as heartbeats, resets the idle timer
was not tested. The docs ask for a `maxIdleDurationSeconds` above the
maximum duration, so it does not matter.

## Decisions not covered by the design note

1. The `suspendCheck` option of `sendJob` is renamed `stateCheck`.
2. `SendJobOptions.windowStartedAt` is removed. Only the state check
before delivery used it.
3. The `assertTimeToDeliver` call at the start of the request step is
removed. It existed because GetMicrovm ran before `sendJob`, and
`sendJob` makes the same check first.
4. The warning after a failed GetMicrovm now says "could not check
whether the MicroVM has ended", not "is suspended".
5. A denied GetMicrovm in a session now always records the step error
type `MicrovmPermissionDeniedError`. Before, the resume-mode path
rethrew the `AccessDeniedException`.
6. Two unreleased CHANGELOG entries of #1178 said that `microvm` uses
`lambda:ResumeMicrovm`. They are edited.
7. The end-to-end function role gains `lambda:SuspendMicrovm`, which
`suspendAfter` needs and the role lacked.
8. A new default end-to-end scenario, `suspend-after-session`. The
logging fetch of the end-to-end function now logs the request path.
9. The `microvm-session` example uses `maxIdleDurationSeconds: 3600`
(above its 900 s maximum duration) and `suspendedDurationSeconds: 600`
(its timeout).
10. `e2e/probe-lifecycle.mjs` still calls ResumeMicrovm. It probes the
service with the operator's credentials, not through the SDK.
11. The `LaunchResult` type drops `resumeBeforeDelivery`. A comment says
that an old checkpoint can carry it.
12. The worker README and TSDoc say that the service resumes the
MicroVM, and that a 503 makes the durable function send the job again.
13. The reserve error of the state check changes its message. Old:
`MicroVM <id> did not reach RUNNING in <n> ms. Last state: unknown.`
New, the same in both SDKs: `GetMicrovm for MicroVM <id> did not end
before the invocation reserve, after <n> ms.` The message can become the
message of the final `MicrovmDeliveryError`, so the checkpoint stores
it.
14. `StateCheckOptions` is no longer exported. Only `lifecycle.ts` uses
it.

## Follow-up

The CI execution role of the integration tests keeps
`lambda:ResumeMicrovm` until both SDKs ship this change. A later change
in the CI infrastructure removes it.

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — 3f441de6 Deployed Nov 7, 2026 by ParidelPooya via ai-pr-review / Codex review / Generate Codex review #2938
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants